Skip to content

A cluster-info outage never stops a managed Worker - #451

Merged
rmcdaniel merged 3 commits into
mainfrom
general/transient-cluster-info
Nov 21, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
general/transient-cluster-info

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

From the final digest re-review. Before this change, any failure of GET /api/cluster/info stopped the PHP Worker.

Why

Every workflow-task completion reads cluster info for the payload policy, at most once a minute (RuntimePayloadUploads::policy()). So does an activity completion that carries payloads. A plain 500 on that read ended Worker::run() before the completion was ever sent. Startup discovery, used for cooperative cancellation and query authority, retried only the narrow untyped-outage set and never reported a lasting outage.

Change

Cluster info is read-only and always safe to repeat. TransientFailure::isReadOutage() treats these as an outage: a refused, reset or timed-out connection, and any 5xx or 429 unless it says retryable: false. 401, 403 and other 4xx answers stay fatal.

  • Run time: RuntimePayloadUploads keeps the last discovered policy through an outage and tries discovery again 10 s later, not before every request. The Server validates every request anyway. With no policy yet, a Worker request fails as a typed payload_discovery_unavailable with request_admitted: false, because nothing was sent. The Worker repeats it with the shared jittered backoff (0.25 s doubling to a 4 s cap) and reports a lasting outage as worker.outage_persisting: once at 60 s, then every 5 minutes. If the completion retries run out, run() keeps going and the Server re-delivers the task when its lease ends. The exact activity-completion discovery envelope keeps its existing handling.
  • Startup: discovery retries read outages the same way, reports a lasting one, and waits the way registration does.

Tests

All of these run over the real Psr18Transport. The failure cases are a 500, a refused connection reported as cURL errno 7, and a bodiless 502. Every one of them fails on main.

  • Workflow-task completion: it waits out 30 failed reads and is sent once. No task failure is reported, worker.outage_persisting is logged, and there is no worker.fatal_failure.
  • Startup: a cooperative Worker started during the outage registers once discovery answers.
  • Stale policy: a unit test covers the fallback. Completions continue with the last policy, and discovery is tried again later rather than before every request.
  • Classification: 5xx and 429 answers and refused connections are outages; retryable: false, 401, 403 and 404 are not.
  • Existing test: testDiscoveryRetryClassificationRequiresAnExactPreSubmissionBackendEnvelope is updated. A 5xx from discovery is now the generic not-admitted outage, still not the exact activity envelope, and a 401 surfaces as itself.

Results: the full suite passes (9229 tests, 56 skipped), and phpstan on src is clean. Docs (docs/portal/build/workers.md) and the CHANGELOG are updated.

The matching PRs are sdk-python#359 (startup) and sdk-rust#306.

Review fix: a stale policy and a lowered limit (commit cf87771)

In review, the Server's inline limit was lowered to 100 KB while the Worker held a 2 MB policy. A 500 KB result went out inline and got 413 payload_too_large. That error was not caught, so the Worker exited.

  • Client: when a completion is refused with a 413 before its body is read, nothing was recorded. The client drops the cached policies, reads cluster info again, encodes the same outcome under the new policy (externalising what it requires), and sends it once more. If re-encoding changes nothing, the refusal stands. If discovery is down at that point, the completion counts as not sent, and the Worker retries it under the outage rules without recomputing the result.

  • Worker: an activity result that is still too large is reported as a non-retryable ActivityResultTooLarge failure, and the Worker keeps running. The cluster-info outage is now tracked separately, so polls that succeed no longer reset it.

  • Tests: WorkerStalePayloadPolicyTest changes the server-side limit mid-run. It covers four cases:

    • the result is externalised and completed once, after one rediscovery;
    • an outage during rediscovery is retried without recomputing the result;
    • an outage that outlasts the claim leaves the attempt to the Server, and the re-delivered attempt completes;
    • a result that is still too large fails the attempt non-retryably.

    Existing genuine-413 tests now expect the single policy re-read.

🤖 Generated with Claude Code

Digest re-review: any failure of GET /api/cluster/info killed the PHP
Worker. Every workflow-task completion (and activity completion with
payloads) reads it for the payload policy, at most once a minute; a plain
500 there ended Worker::run(), and startup discovery retried only the
narrow untyped-outage set without reporting a lasting outage.

Cluster info is read-only, so TransientFailure::isReadOutage() treats a
refused, reset or timed-out connection and any 5xx or 429 (unless
retryable: false) as an outage; 4xx refusals stay fatal.

- RuntimePayloadUploads keeps the last discovered policy through an outage
  and tries discovery again 10 s later instead of before every request.
  With no policy yet, a Worker request fails as a typed not-admitted
  payload_discovery_unavailable (nothing was sent), which the Worker
  repeats with the shared jittered schedule and reports through
  worker.outage_persisting at 60 s, then every 5 minutes. The exact
  activity-completion envelope keeps its existing handling.
- Startup discovery (cooperative cancellation, query authority) retries
  read outages the same way and reports a lasting one; it waits like
  registration.

Tests over the real Psr18Transport (500, refused connection with cURL
errno 7, bodiless 502): a workflow-task completion waits out 30 failed
reads, is sent once, nothing is reported as a task failure, and the Worker
keeps running; a cooperative Worker started during an outage registers
once discovery answers. All fail on main. A unit test covers the stale
policy fallback.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel and others added 2 commits November 20, 2026 11:16
Review blocker: the Server's inline limit was lowered to 100 KB while the
Worker held a 2 MB payload policy. A 500 KB activity result was sent
inline, the Server answered 413 payload_too_large, the refusal was
uncaught and the Worker exited (the activity then failed as having lost
its lease). The stale-policy fallback from the previous commit widens
that window; a fresh 60 s cache already allowed it.

Client: on a 413 payload_too_large for an activity or workflow-task
completion (refused before the body was read, so nothing was recorded),
drop the cached policies, read cluster info again, encode the same
outcome under the new policy (externalising what it requires) and send it
once more. If re-encoding changes nothing, the refusal stands. A
discovery outage at that point fails as not sent, and the Worker repeats
the completion under the outage rules without recomputing the result.

Worker: an activity result that is still refused for its size is the
attempt's failure, not the Worker's: it is reported as a non-retryable
ActivityResultTooLarge and the Worker keeps running. The cluster-info
outage tracker is now separate, so polls that keep succeeding no longer
reset it and a discovery outage spanning several claims still escalates.

Tests (real Psr18Transport, server-side limit change mid-run): the result
is externalised and completed once with one rediscovery; a rediscovery
outage is retried without recomputing; an outage outlasting the claim
leaves the attempt to the Server and the re-delivered attempt completes;
a result too large even under the new limit fails the attempt
non-retryably. Existing genuine-413 tests now expect the one policy
re-read before the refusal stands.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Rust (#306) reports a still-too-large result with failure.type and
failure.class "ActivityResultTooLarge"; PHP sent its fully qualified
class name. Send the same plain string in both fields, so a workflow sees
one reason code whichever SDK ran the activity (Python #360 matches).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel rmcdaniel left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the coordinated worker payload-limit fix against the PHP #451, Python #360 and Rust #306 changes. PHP, Python and Rust recover only from the Server's typed size refusal. The worker can rediscover the policy and re-encode the result without re-running the activity. A result that still cannot fit is reported with type and class ActivityResultTooLarge and non_retryable: true, and worker dispatch remains available. A discovery outage keeps the attempt eligible for recovery instead of recording a permanent size failure.

The target-branch and language CI jobs passed for all three reviewed heads. Python also passed the local focused transport suite, and Rust passed the 17 worker-outage tests. The release qualification and published-package checks follow the merges.

@rmcdaniel
rmcdaniel merged commit 20e5155 into main Nov 21, 2026
37 checks passed
@rmcdaniel
rmcdaniel deleted the general/transient-cluster-info branch November 21, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant